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• IMAGE COMMUNICATION APPARATUS 

The present invention relates to handwriting and draw- 
ing communication devices generally. 

There exists a significant amount of activity in the 
field of on-line handwriting analysis. The prior art current to 
1990 is reviewed in "The State of the Art in On-Line Handwriting 
Recognition" by Charles C. Tappert et al, IEEE Transactions on 
Pattern Analysis and Machine Intelligence, Vol, 12, No. 8, Au- 
gust, 1990. 

Generally speaking, on-line handwriting analysis is 
currently employed for two distinct applications identity verifi- 
cation and input of handwritten letters and numbers into a 
computer. These two applications have sharply contrasting opera- 
tional requirements and goals. Handwriting analysis for identity 
verification senses features of handwriting which are distinct 
for each individual and thus can be used to unambiguously identi- 
fy a given individual. In contrast, handwriting analysis for 
alphanumeric input to a computer seeks to minimize the effect of 
the very features which are important for identity verification 
and to concentrate on universal handwriting characteristics which 
can be associated with given symbols independently of the indi- 
vidual writer. 

Currently existing and proposed systems providing 
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handwriting analysis for alphanumeric input to a computer are 
generally geared towards recognition of how a symbol looks rather 
than how it is created. Accordingly, such systems employ digitiz- 
ers or graphic tablets. 

Signature verification systems, on the other hand, 
attempt to identify biometric characteristics of the writer and 
employ indications such as pressure and acceleration during 
writing. 

U.S. Patent 4,345,239 employs pen acceleration for use 
in a signature verification system. U.S. Patent 5,054,088 employs 
both acceleration and pressure data characteristics of handwrit- 
ing for identity verification. As indicated by the above patents, 
pen acceleration is employed for signature verification because 
it is a personal feature, characteristic of each individual. 
Accordingly, pen acceleration has not been employed for communi- 
cation of hand imaging. 

U.S. Patent 4,817,03 4 describes a computerized hand- 
writing duplication system employing a digitizer pad. U.S. 
Patent 4,641,354 describes apparatus for recognizing and display- 
ing handwritten characters and figures in which unrecognized 
stroke information remains on the display screen. U.S. Patent 
4,715,102 describes a process and apparatus involving pattern 
recognition. U.S. Patent 4,727,588 describes a system for auto- 
matic adjustment and editing of a handwritten t.ext image, which 
preserves format information in a handwritten text. U.S. Patent 
4,703,511 describes a writing input and dynamics regeneration 
device wherein a time dependent code is embedded in a writing 
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The present invention seeks to provide improved hand- 
writing and drawing communication apparatus. For convenience and 
conciseness, the term "hand imaging" will be used throughout the 
specification and claims to denote handwriting activity as well 
as drawing activity and any other two or three dimensional image 
generating hand movements. 

There is thus provided in accordance with a preferred 
embodiment of the present invention communication apparatus for 
hand imaging including apparatus for sensing features of hand 
imaging of an individual which features are highly characteristic 
of the individual but which also contain information relating to 
images represented thereby and apparatus for providing a non- 
individual dependent output indicating the images in response to 
the sensed features. 

The phrase "non-individual dependent output" denotes an 
output which contains non-individual characteristics as well as, 
optionally, individual characteristics. Non-individual charac- 
teristics may include a geometric representation of a hand imag- 
ing product. The non-individual dependent output is typically in 
a form that can be communicated and read universally, for exam- 
ple, by any device which is capable of reading data in a standard 
format. 

In accordance with a preferred embodiment of the 
present invention the apparatus for sensing is contained in a 
hand-held housing. Preferably the apparatus for sensing includes 
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apparatus for communication of the non-individual dependent 
output. 

In accordance with a preferred embodiment of the 
present invention, the apparatus .for communication is operative 
to communicate information which can be used to reconstruct an 
individual's hand imaging style. 

In accordance with a preferred embodiment of the 
present invention, the apparatus for sensing does not require a 
tablet. Alternatively it may include a tablet. Additionally in 
accordance with a preferred embodiment of the invention, the 
apparatus for acquiring and encoding communicates via a modem. 
The communication may be in a fax format or alternatively in a 
compressed non-raster format. Preferably, the communication is 
wireless communication. 

There is also provided in accordance with a preferred 
embodiment of the present invention communication apparatus for 
hand imaging including apparatus for sensing features of hand 
imaging of an individual which features are highly characteristic 
of the individual but which also contain information relating to 
images represented thereby, and apparatus for providing a non- 
individual dependent output indicating the images in response to 
the sensed features. 

Additionally in accordance with a preferred embodiment 
of the present invention the apparatus for sensing features is 
contained in a hand-held housing. 

Still further in accordance with a preferred embodiment 
of the present invention the apparatus for sensing features is 
contained in a tablet assembly. 
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Further in accordance with a preferred embodiment of 
the present invention the communication apparatus also includes 
apparatus for communication of the non-individual dependent 
output. 

Additionally in accordance with a preferred embodiment 
of the present invention the communication apparatus also in- 
cludes apparatus for communication of the non-individual depend- 
ent output. 

Further in accordance with a preferred embodiment of 
the present invention the communication apparatus also includes 
apparatus for communication of the non-individual dependent 
output. 

Still further in accordance with a preferred embodiment 
of the present invention the communication apparatus is operative 
to communicate information which can be used to reconstruct an 
individual's hand imaging style. 

Additionally in accordance with a preferred embodiment 
of the present invention the sensing apparatus does not require a 
tablet. 

Further in accordance with a preferred embodiment of 
the present invention the apparatus for communication includes a 
modem. 

Still further in accordance with a preferred embodiment 
of the present invention the apparatus for communication is 
operative to communicate in a fax format. 

Additionally in accordance with a preferred embodiment 
of the present invention the apparatus for communication is 
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operative to communicate in a compressed non-raster format. 

Further . in accordance with a preferred embodiment of 
the present invention the apparatus for communication is opera- 
tive for wire communication. 

Still further in accordance with a preferred embodi- 
ment of the present invention the apparatus for communication is 
operative for wireless communication. 

Additionally in accordance with a preferred embodiment 
of the present invention the apparatus for sensing features 
includes apparatus for sensing the instantaneous angle of motion 
during hand imaging. 

Further in accordance with a preferred embodiment of 
the present invention the apparatus for providing a non-individ- 
ual dependent output is operative for providing an output indica- 
tion of strokes generated during hand imaging. 

There is also provided in accordance with another 
preferred embodiment of the present invention communication 
apparatus for hand imaging including apparatus for sensing fea- 
tures of hand imaging of an individual which features are highly 
characteristic of the individual but which also contain informa- 
tion relating to images, represented thereby, and apparatus for 
providing an output indicating the images in response to the 
sensed features, and wherein the apparatus for sensing features 
including apparatus for sensing the instantaneous angle of motion 
during hand imaging and providing an output indication of strokes 
generated thereby. 

There is also provided in accordance with another 
preferred embodiment of the present invention apparatus for 
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communicating hand imaging including hand-held apparatus for 
sensing motion and providing an output in a compressed form which 
can be transmitted by a conventional modem, LAN or other communi- 
cations medium. 

Further in accordance with a preferred embodiment of 
the present invention the apparatus includes apparatus for re- 
ceiving communicated stroke content information and being opera- 
tive for reconstructing therefrom hand-imaging information. 

Still further in accordance with a preferred embodi- 
ment of the present invention the apparatus for receiving is 
operative to reconstruct hand-imaging information in three dimen- 
sions . 

There is also provided in accordance with another 
preferred embodiment of the present invention communication 
apparatus for hand imaging including apparatus for sensing motion 
during hand imaging and providing an output indication of stroke 
content in a compressed format, and apparatus for receiving 
communicated stroke content information and being operative to 
reconstruct therefrom hand-imaging information. 

Further in accordance with a preferred embodiment of 
the present invention the apparatus for receiving is operative to 
reconstruct hand-imaging information in three dimensions. 

There is also provided in accordance with another 
preferred embodiment of the present invention a communication 
method for hand imaging including sensing features of hand 
imaging of an individual which features are highly characteristic 
of the individual but which also contain information relating to 
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images represented thereby, and providing a non-individual de- 
pendent output indicating the images in response to the sensed 
features. 

There is also provided in accordance with another 
preferred embodiment of the present invention a communication 
method for hand imaging including sensing features of hand imag- 
ing of an individual which features are highly characteristic of 
the individual but which also contain information relating to 
images represented thereby, and providing an output indicating 
the images in response to the sensed features, and wherein the 
sensing features includes sensing the instantaneous angle of 
motion during hand imaging and providing an output indication of 
strokes generated thereby. 

There is also provided in accordance with another 
preferred embodiment of the present invention a method for commu- 
nicating hand imaging including hand-held apparatus for sensing 
motion and providing an output in a compressed form which can be 
transmitted by a conventional modem, LAN or other communications 
medium. 

There is also provided in accordance with another 
preferred embodiment of the present invention a communication 
method for hand imaging including sensing motion during hand 
imaging and providing an output indication of stroke content in a 
compressed format, and receiving communicated stroke content 
information and reconstructing therefrom hand-imaging informa- 
tion. 
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The present invention will be understood and appreciat- 
ed more fully from the following detailed description, taken in 
conjunction with the drawings in which: 

Fig. 1 is a pictorial illustration of a device for 
acquiring and encoding hand imaging which is constructed and 
operative in accordance with a preferred embodiment of the 
present invention in an operative environment; 

Fig. 2 is a simplified illustration of a preferred 
structure of the acquiring and encoding device of Fig. 1; 

Fig. 3 is a pictorial illustration of another device 
for acquiring and encoding hand imaging which is constructed and 
operative in accordance with a preferred embodiment of the 
present invention in an operative environment; 

Fig. 4 is a simplified illustration of a preferred 
structure of the acquiring and encoding device of Fig. 3; 

Figs. 5A and 5B are pictorial illustrations of a pocket 
held communicator unit and pen incorporating the apparatus of 
Figs. 1 and 2; 

Fig. 6 is a simplified illustration of apparatus for 
communicating and reconstructing three-dimensional hand imaging 
information in accordance with a preferred embodiment of the 
present invention. 

Attached herewith are the following appendices which 
aid in the understanding and appreciation of the invention shown 
and described herein: 

Appendix A is a detailed exposition of the findings of 
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the inventor concerning the characterization of hand imaging via 
pen strokes; 

Appendix B is a computer listing comprising a preferred 
implementation of a portion of the programming of microcontroller 
26; 

Appendix C is an explanation of the computer listing of 
appendix B; 

Appendix D is a computer listing comprising a preferred 
implementation of a portion of the programming of microcontroller 
46; 

Appendix E is an explanation of the computer listing of 
appendix D; 

Appendix F is a computer listing comprising a preferred 
implementation of a portion of the programming of computer 13; 

Appendix G is an explanation of the computer listing of 
appendix F; 

Appendix H is a computer listing comprising a preferred 
implementation of a portion of the programming of computer 13; 
and 

Appendix I is an explanation of the computer listing of 
appendix H. 



11 



WO 94/18663 PCT/US94/01095 

It has been found by the present inventor that each 
writer produces conventional alphanumeric characters as well as 
graphic images, i.e* hand imaging, from pen strokes selected from 
a set containing approximately 12 - 14 pen strokes which set is 
characteristic of each individual. The present invention employs 
this realization to provide communication apparatus for hand 
imaging. Appendix A contains a detailed exposition of the finding 
of the inventor. 

Reference is now made to Fig. l, which illustrates a 
pen for communicating hand imaging 10 constructed and operative 
in accordance with a preferred embodiment of the invention in a 
typical operating environment wherein it communicates by wireless 
communication with a communicator 11, such as a telephone or RF 
modem having an associated receiver, such as a model RB 1023 RF 
receiver, commercially available from RF Monolithics, Inc. of 
Dallas, Texas. Alternatively, communication may be effected via 
any other suitable communications medium such as, for example, a 
local area network and a cellular telephone network. 

The hand imaging communicating pen 10, which is illus- 
trated in greater detail in Fig. 2, may be used on any writing 
surface or alternatively in the absence of a writing surface and 
does not require any special pad or writing substrate. Preferably 
the handwriting input device comprises a housing 12 in the gener- 
al size and shape of an ordinary pen. 

Communicator 11 may communicate via telephone or coax- 
ial cabling or wireless facilities with any suitable receiver, 
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such as a computer terminal 13 having an associated printer 14. 
Thus it may be appreciated that the hand imaging produced by the 
user using apparatus 10 may appear nearly instantaneously on a 
printed hard copy or a computer screen at a remote location. The 
same general type of apparatus may be employed for display on a 
television screen or for output on a facsimile receiver. 

As seen in Fig. 2, disposed in housing 12 is an ink 
reservoir and output point assembly 16, which may be constructed 
and operative in any conventional manner. Alternatively no ink 
output may be provided. In accordance with a preferred embodiment 
of the present invention there is disposed in a forward location 
of the housing 12 an accelerometer 20, preferably operative in 
three dimensions. A typical accelerometer which meets the size 
and power requirements of the invention comprises three mutually 
orthogonally mounted Model 3031-002 accelerometers commercially 
available from EuroSensor of 20 - 24 Kirby Street, London, Eng- 
land. Alternatively, more than 3 Model 3031-002 accelerometers 
may be used. 

The output of the accelerometer 20 is supplied via an 
operational amplifier 24, such as a model LT1179, commercially 
available from Linear Technology Corporation of Milpitas, Cali- 
fornia, to a microcontroller 26, such as an Hitachi H8/536 micro- 
controller including an A/D converter. Microcontroller 26 is 
operative to extract a plurality of predetermined features of the 
acceleration sensed by accelerometer 20. It is a particular 
feature of the present invention that a relatively small number 
of discrete features derived from sensed acceleration during hand 
imaging has been found to be sufficient to map the alphanumeric 
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symbols and graphic output of a given individual. It is appreci- 
ated that the characteristics of such features vary from individ- 
ual to individual and it is often desirable to communicate these 
personal characteristics. Accordingly, the microcontroller 26 is 
programmed in order to preserve and communicate not only the 
information content but also the personal hand imaging character- 
istics of the writer. 

A preferred listing in the C programming language of 
software that provides the functionality of the microcontroller 
26 appears in Appendix B. A brief discussion of the principles 
underlying the functionality of the microcontroller 26 as exem- 
plified in the software of Appendix B, appears in Appendix C. 

The microprocessor 26 is operative to extract a plural- 
ity of strokes and to encode them in a conventional universal 
code, such as ASCII, which is not in any way dependent on the 
personal handwriting characteristics of a given individual and 
which can be readily accepted by conventional computers, modems 
and the like. 

Preferably, the coded symbol output from microcontrol- 
ler 26 is in a form compatible with or identical to the output 
conventionally received from or modem input to a conventional 
computer, such as a PC. 

In accordance with a preferred embodiment of the 
present invention, the coded output of microcontroller 26 is 
transmitted to communicator 11 in a wireless manner by a wireless 
transmitter 32, such as a model MB1003, which is also commercial- 
ly available from RF Monolithics, Inc. and which communicates 
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with receiver 12 (Fig 1). Alternatively any other suitable IR 
transmitter or radio transmitter may be utilized. 

Alternatively a non-wireless communication connection 
may be provided as described hereinbelow with reference to Figs. 
5A and 5B. In such a case, a non-volatile memory such as a flash 
RAM 34 is preferably provided to store the output of the micro- 
controller 26. A suitable battery 33 is provided to power the 
apparatus located within housing 12. 

It will be appreciated that the apparatus of the 
present invention is preferably a hand-held "pen" which can be 
carried by a user and used with any suitable communication facil- 
ities. The communication facilities and computers as well as 
peripherals communicating therewith need not be personalized in 
any way, inasmuch as all of the handwriting recognition hardware 
and software is resident in the "pen". 

Reference is now made to Figs. 3 and 4 which illus- 
trates an alternative embodiment of the present invention which 
employs a tablet assembly 40 instead of pen 10. The tablet assem- 
bly 40, as seen in Fig. 4, may comprise any conventional graphic 
input tablet 42, such as a Summagraphics compatible tablet which 
operates together with a dedicated pen 44 and which outputs x,y 
coordinates and pen lift signals to a microcontroller 46. 

Microcontroller 46 may have all of the relevant func- 
tionality of microcontroller 26 described hereinabove for stroke 
extraction and encoding and may communicate directly with a 
modem/fax unit 48, such as is commercially available from Rock- 
well and other suppliers. A preferred listing in. the C program- 
ming language of software that provides the functionality of the 
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microcontroller 46 appears in Appendix D. A brief discussion of 
the principles underlying the functionality of the microcontrol- 
ler 46 as exemplified in the software of Appendix D, appears in 
Appendix E. 

The output of modem/fax unit 48 may be connected to an 
ordinary telephone or network jack 50 for communication in the 
manner described hereinabove in connection with Fig. 1. 

Appendix F sets forth a software listing in the C 
programming language for reconstruction functionality of the 
information communicated by the tablet assembly 40. This software 
may be resident in computer 13 or alternatively in any output 
device whose function it is to provide a useful output from the 
communicated hand imaging information. Appendix G is a brief 
discussion of the principles underlying reconstruction function- 
ality embodied in the listing of Appendix F. 

Appendix H sets forth a software listing in the C 
programming language for reconstruction functionality of the 
information communicated by the pen 10. This software may be 
resident in computer 13 or alternatively in any output device 
whose function it is to provide a useful output from the communi- 
cated hand imaging information. Appendix I is a brief discussion 
of the principles underlying reconstruction functionality em- 
bodied in the listing of Appendix H. 

Further in accordance with a preferred embodiment of 
the present invention there is provided portable information 
storage and retrieval apparatus including a portable computer 
memory and output device and having as an input element hand-held 
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apparatus of the type described hereinabove. 

Reference is now made to Figs. 5A and 5B which illus- 
trate acquisition and encoding apparatus for hand imaging and 
communication apparatus in accordance with an alternative embodi- 
ment of the present invention. Here a communicator 100 is formed 
with a socket 101 for removably accepting acquisition and encod- 
ing apparatus 102. Data communication contacts 104 and 106 are 
disposed respectively at an end of the apparatus 102 and in 
socket 101 for permitting downloading of written information from 
apparatus 102 to the communicator 100. Preferably the communica- 
tor is designed to be pocket sized. 

Reference is now made to Fig. 6, which illustrates the 
use of the apparatus of the invention for communicating three 
dimensional information. A user employing the apparatus outlines 
the shape of a three dimensional object, such as an airplane 
model. The user may follow the lines of an existing three dimen- 
sional model, as in the illustration, or alternatively, may draw 
in a free hand manner. The hand imaging produced by the user is 
communicated via communicator 11 to a communicator 120 and thence 
to utilization apparatus such as a CNC machine 122 or a three 
dimensional model building machine 124, such as is commercially 
available from Cubital Ltd. of Ramat Hasharon, Israel. 

Communication of hand imaging information in accordance 
with the present invention may be in two different modes. Where 
compression is desired along the communication channels, the hand 
imaging information may be transferred in a penstroke language. 
In such a case, a hand imaging reconstructor is required at the 
remote location. 
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Alternatively, the hand imaging information can be 
reconstructed upstream of communicator 11, by suitable recon- 
struction apparatus. In such a case, the hand imaging information 
may be communicated in conventional CAD format. 

It will be appreciated by persons skilled in the art 
that the present invention is not limited by what has been par- 
ticularly shown and described hereinabove. Rather the scope of 
the present invention is defined only by the claims which follow: 
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What the human brain tells the human 

hand 

A behavioral perspective 

Ehud Bar- On 
E.B. Research L Development Ltd. 
Guttwirth Bldgs. Technion. Haifa 32000, Israel 

September 22, 1992 
Abstract 

This study investigates the primitives of motoric patterns of hand 
movement during handwriting. This is referred to as the "language" 
between the 'hand and the "brain", and as such, has its own vocabu- 
lary and syntax. The "vocabulary" is the pen strokes and the syntax is 
how they are combined to pen strokes sequences. The handwriting is 
viewed as a high level cognitive activity of communicating , expressed 
as a complex motor skill and its investigation provides insight into the 
processes of chunking and automaticity. The main finding of this study 
is that pen strokes are specific to an individual writer, and character- 
ize the writer : s unique motoric control mechanism. The dynamic data 
of many thousand handwritten characters t produced by many writers, 
had been analyzed. The time domain signals were segmented into dis- 
crete pen strokes units and represented as vectors in a feature space. 
Those vectors were clustered, using a variety of clustering techniques. 
We found that in spite of the fact that the hand movements during 
writing could take any form or shape, a particular writer employs only 
a very limited set of pen strokes. The results of the clustering by var- 
ious methods, yields a limited set of only twelve to fourteen types of 
pen strokes that accounts for 90brain supposedly chunks information 
to minimize the required attentional resources. 
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1 Humans motor behavior 

VVheu we speak about "what the human brain telis the human -2nd", we 
speak about the kind of "motor control language", that might exir. between 
the "brain" and the ""hand' 1 . While this ''brain-hand" communis: ion can 
be approached from different points of view, we study it from a behavioral 
perspective. That is. we investigate the evidence of chunks or what is referred 
to as "motor programs", by analyzing the dynamic data collected during 
experiments in hand-writing recognition. Although all the results reported 
in this article consists only on the output of the handwriting process, we try 
to make the proposed model cognitive and biological plausible. * 

We'll start with the cognitive plausibility. There is a strong link between 
cognitive mechanisms and the human motor behavior. Handwriting is the 
way that humans express their thoughts through the use of a complex motor 
skill. Rosenbaum [12], presented handwriting as culmination of several in- 
ternal translation process. First, an abstract message or idea is constructed. 
Then, it is formulated into appropriate linguistic expression, and then trans- 
lated as a series of efferent commands. There is a basic similarity between 
speech and writing, so we can assume that both share the same underlying 
mechanism. The phonemes in speech, correspond to pen-strokes in writing, 
and the morpheme in speech correspond to letters. The higher levels of ab- 
straction (i.e. words, syntax, lexicon, semantics, prosody and discourse), are 
probably the same. 

There is an empirical evidence, that the ability to sequence behavior, 
whether in the linguistic domain or drawing domain, depends on a central, 
amodal mechanism [11]. If this is true, then motoric control should be consid- 
ered as obeying the same rules of the linguistic or other sequential cognitive 
mechanisms. On the other hand, some of the properties that we discover 
about the primitives of the motor control language, might be generalized to 
other cognitive activities. In the case of handwriting, the efferent commands 
are expressed as pen-strokes. It was found by Wright [17], that different pro- 
duction mechanism are probably controlled by the same high-leve*. zraphic 
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representation. This view of hierarchical structuring, and a "virtual" repre- 
sentation of motor movement is supported by empirical evidence. Humans 
can write in a consistent style when they write in small letters in :~eir note- 
books, or when writing much bigger characters on a blackboard. Moreover, 
people can write with a consistent style (same pen-strokes), when using dif- 
ferent effectors like hand and foot. 

As in any behavioral research, resulting behavior is influence*: both by 
the general properties of handwriting and by properties which are specific 
to an individual writer. This means that a considerable part of the variance 
can be attributed :o individual differences. It has been noticed by many 
researchers that handwriting style is so distinctive, that writers can be rec- 
ognized according to their hand writing. This is also a common knowledge, 
and therefore signatures are recognized as a unique identifier of a specific 
writer. Some theories even associate personality traits with hand-writing 
style. As we intend to show in this article, the primitive pattern of writing, 
are unique for individuals writers. 

As any other cognitive or motor activity, human motor conrroi goes 
through a process of development that is equivalent to intellectual develop- 
ment. An interesting well known phenomena is that children's drawing (and 
after that writing), becomes more refined over the course of development. It 
has been suggested by many researchers, that early drawing behavior corre- 
lates with young children's cognitive abilities. Van sommers [15]. suggested 
that drawing may be governed by high-level rules, similar to those governing 
language processing, and that the development in drawing may parallel the 
development of language. Goodnow and Levine [9] even suggested a: " Gram- 
mar for action: sequence and syntax of children's copying." They reported 
several rules for sequencing drawing strokes. Examples of such ruies were:" 
Start at leftmost point", "Start at top", "Start with vertical strokes", "draw 
horizontal lines from left to right", etc. The evolutionary rationale for such 
rules could be to simplify motor planning. 

The biological plausibility of an hand writing model, involves two parts: 
The plausibility of the assumed neurological control, and the biomechanical 
properties of the hand. The preservation of the writing style while using dif- 
ferent muscles and even organs, is one of the most intriguing questions. The 
automaticity of writing, suggests a chunking mechanism, but this chunking 
mechanism is probably not in the motoric system of the hand, but somewhere 
in the "upper control levels of the brain. Therefore, whenever we refer to 
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the "hand". we dc :: metaphorically. I.e. ihe "hand" represents the efferent 
mechanism that accomplishes the motoric control. Recently, Alexander et al. 
[3], raised the question whether the specific concept of a "motor program", is 
an appropriate foundation for the development of biological plausible models 
of how the brain controls movements. While our current knowledge about 
the cortical and basal ganglia motor areas is still far from allowing specific 
model, it can suggest what modeis are more neurologically plausible than 
others. Fischbach 7; discusses the finding about "face cells" and "motor 
command cells" as an evidence for abstraction in the brain. In the monkey's 
visual system, "face cells'" located in the inferior temporal sulcus. *-vere sug- 
gested as representing a high level of abstraction. These neurons respond to 
faces but not- to other visual stimuli. Face cells have their counterpart on the 
motor side. "Command" neurons have been identified in certain vertebrates 
that trigger fixed action patterns. Georgopolos [1] recorded electrical activ- 
ity of single neurons, and found command neurons in the monkey's motor 
cortex (precentral gyrus) that encode the direction of forelimb movement. 
The firing of these neurons was not associated with the contraction z-i a par- 
ticular muscle or with the force of the coordinate movement. Georropoulos 
computed a vector by summing the firing frequencies of many neurons, and 
found that it is more correlated with the direction of movement than is the 
activity of any individual cell. The vector becomes evident several millisec- 
onds before the arm moves. He interpreted this result as evidence for motor 
neuron planing. Damasio and Damasio [4] discussed the linguistic behavior 
of patients with lesion in the left posterior temporal and inferior parietal cor- 
tex. It was found that such patients have problem in producing word forms 
from the available phonemes. Analyzing the accumulated empirical finding 
on language structures, gathered with assistance of imaging techniques like 
MRI (Magnetic Resonance Imaging) and PET (Positron Emmisioc Tomog- 
raphy), shows that linguistic activity like naming, involves the motor cortex 
activation together with anterior and posterior language centers in the left 
hemisphere. Writing is a language activity which involves a production cen- 
ter that forms words and activates the "command cells" in the motcr cortex 
to produce pen-strokes sequences (letters) and written words. In the same 
jway that speech is composed of a small set of phonemes, we argue that 
handwritten letters are composed of a small set of pen-strokes. 

In addition to the neurological plausibility, there are biomech-nic con- 
straints on the "hand" part. Some general principles have been suggested 
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as governing this control mechanism. For example. Flash and Kc-gan [Sj. 
proposed that humans tend to write in a way that minimizes jerk. That is, 
the third time derivative of the position signal. A more recent study [6] : 
suggested the snap, which is the fourth derivative of position, as :he cost 
function that is minimized. As we will see, there are alternative hypotheses 
about the type of constraints imposed on the biomechanics of hancwriting. 
In spite of the fac: that the principle that governs :he handwriting might 
be universal, each writer has its own unique variation. The differences are 
more pronounced in the unwritten strokes (the pen movements the: do not 
touch the writing surfaced than in the written ones. The friction o: :ae pen 
with the writing surface diminishes the characteristics of the han-i control 
mechanism which are better revealed when the pen is up. 

The article starts with reviewing theories of "motor programs", and ar- 
guing against thai term and what it implies. We'll propose an alternative 
connectionist model of primitive hand-writing patterns and argue :hat it is 
more biologically and cognitive plausible. Then, we describe the experiment, 
and the collection of the data. The fourth chapter will describe the analysis 
of the collected data, and the conclusions that were drawn from this analy- 
sis. The last chapter will discuss the results, and compare our conclusions to 
alternative ones. We shall conclude the article by pointing out some future 
directions and implications of the suggested model. 

2 Attention, chunking and "motor programs" 

The concept of "working memory" is modeled after the "working memory" 
in a (von Neumann) computer, where the registers in the Central Processing 
Unit (CPU) have a similar function. This is also why researchers in that field 
prefer to talk about "motor programs"'. "Motor programs" are supposed 
to save attentional resources. According to this approach, it is assumed 
that the brain controls movements like handwriting, by executing "motor- 
programs'', much like software is used in a computer (e.g. [12] ). The "motor 
program" concept is attractive, as it reduces the complexity of the sequential, 
analytical approach by using pre-programmed sequences of a limited number 
of generic motor commands (or routines), to control a large repertoire of 
movements. Alexander et al. [3] points out difficulties with the neurological 
plausibility of n motor programs" that imply separation between ~sc:r.vare" 
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and "hardware", ror example, what would constitute the software in such 
a model, and where it is stored when not executed, how are they assembled 
prior to their execution and how new programs are created. A major problem 
with the "motor program" approach is also the sequencing of performance: 
goal- directed movements are supposed to be translated into trajectories, 
then to joints kinematics. Muscle activation cannot be computed until the 
inversedynamics is calculated and so on. Therefore, argue Alexander et al.. 
signs of specializazion for such transformations should have been round in 
the cortical and basal ganglia. Thusfar. neurobiological evidence seems to 
indicate lack of such specialization. 

It is assumed that the processing capacity is limited, and therefore sev- 
eral tasks that have to be carried out simultaneously compete on the same 
resources. The main "problem" 1 of human-beings and other organism might 
be. that we lack a "parallel output channel". All the output chancels, be it 
speech, handwriting or any other motor output, are ail serial in nature. It 
might be that this serial output suggested a serial cognitive mechanism as 
well. It is the conjecture of the neural-nets literature, and of this anicle. that 
the underlying mechanism is parallel and distributed over millions of simple 
processing units (neurons). Therefore, the term "motor programs", that im- 
plies a serial symbolic process running in the "brain- computer" might be 
misleading. We prefer to speak about "motoric schemas". which are motoric 
patterns invoked by activation of an assembly of neurons. 

The connectionist view of schemas (Smolensky. 19S6) is that stored knowledge- 
atoms are dynamically assembled at the time of inference, into context- 
sensitive schemata. Rumelhart and McClelland (19S6) [14] proposed a tech- 
nique that suggests how an attentional selective mechanism might work. 
They propose the use of a set of mapping units which produce "dynami- 
cally programmable connections" and achieve focusing on difTerent features 
on different times. Smolensky (19S6) maintains that schemata are coherent 
assemblies of knowledge atoms, where coherence or consistency is formalized 
under the name of harmony. He proposes the harmony principle: the cogni- 
tive system is activating coherent assemblies of atoms, and draws inferences 
that are consistent with the knowledge represented by the activated atoms. 

In much the same way, we propose to speak about "motoric schemas". 
This is consistent with our conjecture that there is no essential difference 
between the so-cai:ed " cognitive" and "motoric" brain mechanism. The con- 
nectionist schema- model is also consistent with the neural evidence, that 
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the specialization among different cortical motor areas are related :o certain 
sequences of movements, and not to transformations as proposed by :he ''mo- 
tor program" literaiure. According to our conjecture, preparatory units and 
movement executing units will belong to the same schema. This is supported 
by the anatomical fact that the three motor areas (SMA - Supplementary 
Motor Area. PMC - Primary Motor Cortex and Putamen), has ;he same 
proportion of targe: dependent motor cells and limb dependent rr.ovement- 
related cells ((2] ). Another supporting evidence is that neuronal population 
that were supposed :o represent different stages of computation (according 
to the "motor program" view), have been shown to be active simultaneously. 

3 The experiment 
3.1 Data collection 

Rumelhart [13] developed a system which learns to recognize cursive script as 
it is generated by a *.vriter. This system learns from examples of cursive script 
produced by a number of writers and recorded. He collected approximately 
1000 words from each of 5S writers. The average length of a word is about 8 
characters. That sums up to nearly 500,000 examples of handwritten cursive 
characters. His results were encouraging' and had been used in this research. 
While Rumelhart [13] was mainly interested in handwriting recognition, this 
article uses the same data to investigate the writing mechanism. 

The data were collected in the following manner. Each word in the corpus 
was recorded. It was then played to the writer who was instructed to write 
the word on a tablet digitizer. The resulting x coordinate, y coordinate and 
an indication of whether the pen was or was not on the paper were sampled 
each 10 milliseconds. The resolution ( more than 200 dpi) and the sampling 
rate (100 samples /sec) are those that are shown to be appropriate in the 
on-line hand-writing recognition literature ([16] ). The data was saved as 
files, and has been used for the analysis reported in this article. 

In addition to the data from Rumelhart's experiment, several thousands 
pen strokes of Japanese handwriting were collected. Most of the data has 
been collected from hand written Hiragana characters, but some data has 
been collected during writing Kanji (idiographic) Japanese characers. Hi- 
ragana characters has the curved shapes of english hand printed characters, 
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but without the ligature of cursive handwriting. 

Preprocessing of the hand-writing raw data has been made, with the 
goal of extracting features that will be used to segment and characterize the 
"* pen-strokes". A pen stroke was defined as a segments of the cursive writing 
signal, between two consecutive zero crossing of the vertical velocity of the 
pen movement. Each character was segmented to several segments or "pen- 
strokes'. A typical writing rate in English is two letters per second. Writing 
Japanese characters (Hiragana). takes about the same time, and a typical 
Hiragana character can be written in 0.3 - 0.5 seconds. 

3.2 Segmentation and feature extraction 

The principle of segmentation and feature extraction is to segmen: the con- 
tinuous signals into discrete segments and to represent each segment by a 
feature vector in the feature space. 

The segmentation that produces n pen-strokes *' out of the continuous sig- 
nals, depends on the different definitions of the term* 1 pen-strokes". While 
most of the literature about on-line character recognition is using this term, 
there isn't an agreed upon definition of a "pen-stroke n . For example, one 
often finds only the pen-state change as the only criteria. That is. defini- 
tion of a stroke as continuous pen-movement, between pen-down and pen-up 
consecutive states [16]. 

Once a "pen-stroke'' is defined, there are many ways to represent it in 
a feature space. The on-line character recognition research employs several 
orthogonal transformations such as a discrete Fourier transform of :he curve 
segments corresponding to the pen-strokes. That is. a pen-stroke is repre- 
sented by its Fourier coefficients obtained from its x(t) and y(t) signals. Es- 
sentially, any orthogonal transformation (e.g. Walsh transform, Karhunen- 
Loeve) could do in approximating the pen-strokes curves. That is. Plane 
curves can be approximated by orthogonal functions (Sinusoidal, polynomial 
or even square waves). This description can be also easily converted to the 
frequency domain, as was done in several studies of hand-writing recognition 
[16]. 

This mapping of the time domain to a parametric domain is advanta- 
geous when the characters can be represented by a small number of coeffi- 
cients. Therefore, periodical smooth curves lend themselves better to mod- 
eling by harmonic functions, as one needs less coefficients. On ;he other 
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hand, straight line strokes require high order harmonies as they include high 
frequency components. This is why sinusoidal approximation is useful for 
characters consisting of curved strokes, as found in English cursive script, 
more than for Japanese Kanji characters (that are made mainly of straight 
line segments). K-L expansion has been proved to be a successful algorithm 
in machine-printed Chinese character recognition. Another successful at- 
tempt was to use a modified Hough transform for recognition e: Chinese 
hand-written characters. The Hough transform is a technique for Lize detec- 
tion and has been generalized to detect arbitrary shapes. Chinese characters 
are line-like, and cherefor lend themselves naturally to a Hough ::ansform 
representation. 

The segmentation and feature extraction methods depend of course, on 
the goal. If the goal is pattern recognition, then the segmentation and feature 
extraction are geared toward discrimination between the various patterns. In 
our case, we looked for a segmentation and features that are biological plau- 
sible. Consequently, we investigated only features that might be explained 
by the neurobiologies! control structures, like the direction of the strokes, 
their curvature etc. 

3.3 Hollerbach's model 

The segmentation and feature extraction mechanism employed was. to de- 
velop a model of the underlying handwriting process and to describe the data 
in terms of the parameters of the model. The model employed was derived 
from that of Hollerbach [10] and involved the assumption that the genera- 
tion process could be described as pair of coupled oscillators. The coupled 
harmonic oscillators is just one of the many models that exist. Actually, 
its basic assumption about the symmetric shape of the velocity profile (an 
half sinus shape), is probably an oversimplification. The literature about 
velocity profiles of pen-strokes usually assumes an asymetrical beli-shaped 
velocity profile. That is, a rapid-aimed movement described by a log-normal 
velocity profile is considered as the fundamental unit (stroke). More com- 
plex movements are described in terms of superimposed log-normal curves. 
The asymmetric nature of the velocity bell-shaped profile results from the 
global stochastic behavior of a large number of processes involved ir. velocity 
control. 

In spite of being an oversimplified and inaccurate model, it has a clear 
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advantage that it if based on a control mechanism, and is neuqrobic'.ogically 
imerpretable. Thii model assumed that: 



x = a eos(w r * io)4-c 
y = bcos(vyt) 



(2) 



In words, the idea is simply that writing involves two orthogonal pendular 
movements. If we speak about writing in a notebook (small size letters), we 
can think about the wrist horizontal movements (actually, it is more arc-like 
movements) and the fingers flexion andextension vertical movement. These 
two movements czz be considered as independent. If the size of the letters 
is more than an intch. than the arm muscles are involved. 

According to this model. The y-axis consists of a series of up/ down strokes 
whose velocity prof.ie is assumed to be sinusoidal. The x-axis is also pendular 
with a constant velocity, c, to the right. Different characters are made by 
modulating the relative amplitudes, a and 6, the relative phase, phi. and the 
relative frequency _- r and u; y , in the x and y directions. It is, furthermore 
assumed that the parameters change only when the velocity in the y direction 
reaches zero (end of pen-stroke). Thus, we define a stroke as the motion 
between zero crossings in the y velocity - v y . In addition, segmentation 
occurs when the pen-state changes (from pen-down to pen- up or vice versa). 

It should be stressed that Hollerbach's model was designed for synthesiz- 
ing handwritten-like character, by a second order mechanical system. This 
model does not try :o imitate the human motor control, or to be used for 
analysis of human handwriting. However, as it is a control system model, 
some of the parameters might be interpreted in terms of the human biome- 
chanical system. For example, the parameter <p, which designates the phase 
shift, can be interpreted as relating to the delay in the nervous-muscular 
control system. As such, it can have an important diagnostic value in motor 
diseases. 

When it was applied by Rumelhart to handwriting analysis, it suffered 
from some drawbacks. One of them is that the model is fitted not to the 
image, but to the velocity profile of the stroke. This simplifications tend 
to work well in most of the cases of English cursive hand- writing, because of 
the periodical nature of the v y velocity signal. 

As the Hollerbach model that we used, is based mainly on the velocity sig- 
nals, we will illustrate the transformation from the x - y domain of the hand- 
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Figure 2: The v £ graph for the handwritten letter d 

written character, to the corresponding velocities. Examples of the handwrit- 
ten letter d, the corresponding velocity profiles and the reconstructed d are 
shown in Figures 1,2.3 and 4. 

As can be seen, the reconstruction isn't perfect, and the curvature of the 
first pen-stroke of the d is opposite to the original. This result illustrates the 
fact that the model tries to reconstruct the velocities and not the resulting 
pen strokes image. » 

This does not exclude the fact that sinusoidal approximation worked for 
Rumelhart in recognition of cursive script. It turned out that in some cases 
(periodic signals during cursive handwriting in English) the model worked 
satisfactorily. 

The Kanji characters, on the other hand, have more short straight seg- 
ments, as can be seen in the following figures: . 

The "mori" Kanji character in the picture, is segmented to 27 pen-strokes 
(the last two pen-down strokes in the third "tree* 1 are missing). Sixteen out 
of the twenty seven, are strokes in which the pen touched the paper, and 11 
were just for moving the pen from one line to the other. Twelve sequences of 
"pen-down" strokes, correspond to the visible line segments in the character. 



29 



WO 94/18663 



PCT/US94/01095 



1 

•k 

i 


. A 


1 

i 

1 / 


•* 


M 

i 


V 





Figure 3: The v y graph for the handwritten letter d 




Figure 4: The reconstructed letter d 
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Figure 5: The separate strokes axe more evident in Japanese Kanji characters. 
This is the Kanji character: "mori". which means: forest 
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Figure 7: The reconstructed Kanji character: "mori".. 
The reconstructed Kanji character is depicted in the figure. 

3.4 Recognizing pen-strokes sequences 

One of the key problems in recognizing cursive handwriting is the segmen- 
tation problem. Rumelhart [13] has devised a learning algorithm for cursive 
handwriting recognition which combines word recognition and letter recog- 
nition. The letter recognition is based on recognizing PMPs, and PMPs 
sequences make letters. This system involves simultaneously learning to rec- 
ognize and segment letters. 

Although Rumelhart's experiment was done for hand- writing recognition, 
there are several things that can be learned from it. concerning the PMPs and 
their sequencing during handwriting. It was recognized in the early sixties 
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([6]}. that, motor knowledge can be used in recognition of hand-writing. A 
system called Analysis by synthesis suggests that characters are recognized by 
their "motor programs' 1 . These r motor programs" are supposedly deduced 
by guessing an initial program, and iteratively updating it according to the 
difference between the synthesized and actual forms. The connection between 
reading and writing process have been corroborated by the co-occurrence of 
certain "kinds of acquired dysgraphia and dyslexia ([5]). In contrast to ear- 
lier approaches. Edelman et al.. ([6]). assumes that while readers use motor 
knowledge in reading, they do not seem to do so by mentally reproducing 
the process of writing. The connectionist model that we propose isn't both- 
ered, of course, by those distinctions between explicit simulation or implicit 
knowledge. This is another example of the misleading influence created by 
the r motor programs" metaphor. 

4 The results of clustering 
4.1 fixed radius clustering 

The basic units of clustering were the pen-strokes, each of which was rep- 
resented as a point in an n dimensional space. Out of the six features that 
we extracted for each stroke only three have been used. First, we used only 
one frequency for the modeling, so the rare strokes that involved iigher har- 
monies were removed. Second, we did not differentiated between Up-strokes 
and Down-strokes. Up strokes contain more high order harmonies, but we 
limited our analysis to the basic movements, and tried to ignore the fluctu- 
ation induced by the bio-mechanical control mechanism. The third feature 
that wasn't used was the mid-point. For the reconstruction of the pen-strokes 
in the spatial domain, the x-coordinate of the midpoint in each stroke was 
computed. However, our preliminary analysis showed that this variable was 
very highly correlated with the x variable. This preliminary analysis, yielded 
three variables that were almost uncorrected: A x ,Ay and velocity. The 
dimension of the space were: 

.1. A y - The relative displacement on the vertical direction. 

2. A x - The relative displacement on the horizontal direction. 
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3. v - v The v : velocity at the end of the stroke is calculated according to 
equation beio-.v. 

For a pen-stroke between a and b. which is approximated by a certain 
oscillation frequency, we calculate: 

/, = jf or cos(2u; r 0 (3) 
, = ( £Lp M 2j-j ' (4) 

(5) 

It is calculated in a different way for different u:t and this is an example 
of such a calculation. 

The clustering of data from many writers, didn't yield satisfactorily clus- 
tering, but the clustering of individual writers did. E.g. the clustering for a 
particular writer, revealed 13 compact clusters that contained 90to 14. were 
consistent to all the writers we analyzed. This is a corroboration to our con- 
jecture that hand writing is made out of a small number of PMPs. which are 
unique to an individual writer. 

The centroids of the clusters, were reconstructed from the feature space, 
and are displayed on a 2-D spatial domain. As can be seen clearly from the 
results, different writers have different stroke types: 

As each writer has about 25.000 pen-strokes that we wanted to cluster, 
we started with a fast clustering algorithm, similar to the k-means algorithm. 
The main requirement of the clustering algorithm were that it will be able 
to deal with very large data sets and find satisfactory clusters in few (2-3) 
iterations. The other requirement, which was even more important, was that 
the centroids will be good representations of the observations within each 
cluster. This requirement lead to seeking compact, hyperspherical clusters, 
that do not exceed a predefined radius. Elongate clusters are therefore repre- 
sented by several adjacent clusters. Those clusters will be merged in a latter 
stage by an hierarchical clustering algorithm. 

The clustering employed a two phase strategy. First, a fast "nearest 
centroid sorting" algorithm was employed to reveal the clusters in :he large 
data set. Then, the resulting centroids of the clusters have been submitted 
to different hierarchical clustering methods. The first phase algorirhm was 
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Figure S: Clustering of 25,000 strokes of the same writer. Gray clusters 
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Figure 9: Thirteen centroid pen-strokes of an individual writer, including 
their relative frequencies. 
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sensitive to outlier strokes, that formed separate clusters. This was the reason 
why we got many very small clusters. These clusters accounted for less than 
lOof the observations. They were considered to be noise, or very exceptional 
pen strokes, and have been removed so not to influence the representativeness 
of the centroids of the large clusters. 

The second phase included clustering of the resulting centroids using ten 
different methods. We distinguished between methods that yield compact 
hyperspherical clusters, and those that can detect elongate clusters. We 
start with the first group of eight clustering methods: 

1. Average Linkage cluster analysis 

2. Centroid hierarchical cluster analysis 

3. complete linkage cluster analysis 

4. Equal variance maximum likelihood method 

5. Flexible data cluster analysis 

6. McQuitty's similarity analysis 

7. Median Hierarchical cluster analysis 

S. Ward's minimum variance cluster analysis 

The different methods tend to favor different characteristics such as size, 
shape or dispersion. For example, methods based on the least-squares cri- 
terion such as k-means or Ward's minimum variance method, tend to find 
clusters with roughly the same number of observations in each cluster. Aver- 
age linkage is biased toward finding clusters of equal variance. Most cluster- 
ing algorithms, except for single- linkage and density-linkage, tend to produce 
compact, roughly hyperspherical clusters. The clustering methods which are 
based on nonparametric density estimation, like the single linkage, will be 
discussed later in this chapter. 

All the above clustering methods yielded very similar results, and the 
tree-based partition was essentially the same.. The use of many different 
algorithms has been employed to investigate the robustness of the clustering 
structure under different hierarchical clustering methods. The result of all 
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Figure 10: Hierarchical (compact) clustering of the 1*2 pen-strokes centroids 
of a particular writer 



the above method revealed the following tree-based partition of the set of 
the basic twelve pen-strokes (of a particular writer). 

From looking at the results of the hierarchical clustering, there is an ob- 
vious super-clusters that emerge. The horizontal-left strokes are one such 
a group, long down strokes are another group. In general we see a distinc- 
tion between horizontal strokes and vertical strokes. The horizontal strokes 
themselves are subdivided to horizontal-left directed strokes, and horizontal 
right and up directed strokes. The high velocity C shaped strokes are part 
of circles or ovals. It should be noticed that for a specific writer, a certain 
stroke is always accomplished in the same way. For example, an horizontaJ 
short stroke, like crossing a t, will be done always as left directed strokes. 
Someone else could use only horizontal right directed strokes for that pur- 
pose. However, it is very unlikely that the same writer will use both an 
horizontal-left and horizontal-right strokes. The same is true with long ver- 
tical strokes. Once the writer is using a long vertical down-stroke, he will 
produce vertical lines always as down strokes of the same type and velocity 
profile. This organization of pen strokes was consistent in all the hierarchical 
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Figure 11: Hierarchical (Density linkage) clustering of the 12 pen-strokes of 
the same writer 

clustering algorithms that we mentioned above. 

The clustering methods that employ nonparametric density estimation, 
like the "Density linkage cluster analysis", can detect also elongated cluster 
shapes. These clustering techniques yielded two distinct super clusters: the 
"down and long pen-strokes'" , and the "up and right strokes". The down 
strokes are those that form the "back-bone" of the English characters, while 
the up-right strokes are typically those that are used as ligature. 

4.1.1 The "characteristic" shape of pen-strokes 

As was argued above, any writer has a specific set of pen strokes that char- 
acterize the writer. While the same writer will have similar pen-strokes, in 
writing different languages, the frequency of appearance of a specific pen 
strokes depends, of course, on the language. In order to characterized a spe- 
cific writer, in respect to her/his pen strokes, we suggest the "Pen-strokes 
Ordering Diagram" (POD). Such PODs are displayed in the following figures. 

In spite of their strange looking, those diagrams are quite valuable, and 
convey important information about the handwriting of the analyzed writer. 
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Figure 12: The centroids of the pea strokes of a writer, for English cursive 
writing. The pen-strokes are ordered according to their v v values, from up- 
strokes to down strokes 




Figure 13: The centroids of the pen strokes of a Japanese writer, for Japanese 
Hiragana characters. The pen-strokes are ordered according to their v y val- 
ues, from up-strokes to down strokes 
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Figure 14: The centroids of the pen strokes of a Japanese writer, for english 
characters. The pen-strokes are ordered according to their v y values, from 
up-strokes to down strokes 

There is a very clear distinction between the key strokes that these two 
writers are using. This is true to other writers as well. Each writer uses 
a unique set of pen-strokes: different slopes, different curvetures. different 
velocity profiles and accelerations. 

5 Discussion and future research 

We will start our discussion with comparing the conclusions of Rumelharvs 
handwriting recognition experiment, and the conclusions of this study. In 
Rumelhart's handwriting recognition experiments, both writer dependent 
and writer independent recognizers have been trained. Two networks have 
been trained to recognize the writing of individual writers and one network 
has been trained on four different writers as a "writer independent" recog- 
nizer. On the writer dependent networks Rumelhaxt found that, for a vo- 
cabulary of 1000 words, on words never seen during training that 99top five, 
approximately 90On the writer independent data the results are somewhat 
worse. That is, about 70 

According to the results in this study, we have a basis to doubt this 
conclusion. The inter-writer variability is too big, and more writers will not 
lead necessarily to better results. 

Another finding of Rumelhart was that writers can be trained easily to 
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produce recognizable hand writing. He developed an "online" system in 
which the network recognizes (and can be trained) as the writer writes on 
the digitizer. With a little care on the part of the writer it is not circuit to 
achieve a score of better than 90correctly classified on the writer independent 
svstem. (It is also possible to write so that the recognizer does much more 
poorly. than thai. Careful experiments on a person's ability to adjust to 
the recognizer have not been carried out. By limiting the vocabulary to 
one hundred words or less, it seems to be possible to obtain nea: perfect 
performance. (It wiii. of course, depend on the confusability of the words.) 

The main conclusion was. that it would be useful to embed the recognizer 
in a network of networks each trained on a subset of the writers - perhaps one 
for printers, one for pure cursive writers etc. This line of thought led to the 
current study, reported in this article. That is. that it may be useful :o study 
the individual differences among the writers. The idea of studying individual 
differences, as a mean towards better handwriting recognition, turned out to 
start a new line of research - the study of writer's unique pen-strokes, which 
is related to the topic of automaticity in brain - hand communication. 

This study started from that point. The main question that we posed 
was if individual writers have distinct sets of pen-strokes, which are con- 
sistent and well defined. The reanalysis of the data from this perspective 
encourage to believe that this is the case. Human writers have 12- 14 distinct 
pen-strokes, which are characteristic for a certain writer. These pen-strokes 
are the primitive "motoric patterns", of which handwriting is composed. We 
showed also that the primitive pen-strokes cluster to super-clusters, thus re- 
vealing the hierarchical nature of the control mechamnism. These findings 
are consistent with the neurological literature, that we cited in the introduc- 
tion. That is. there might be "command cells*, that get the activation for 
certain words (letter combinations) from another center in the brain and acti- 
vate pen-strokes mechanism. The pen-stroke is controlled by a direction and 
amplitude cells, that activate the corresponding primitive motoric patterns 
(PMPs). The next stage will be to locate the cell regions that are responsi- 
ble for this activation in the motor areas of the human brain. This is under 
research now with the help of Magnetic Resonance Imaging (MRI: method, 
when the MRI is tuned to detect cerebral blood flow. We would expect that 
the learning to write, should show itself as forming of such motoric acrivation 
centers, corresponding to what we have found in this study. 

"Those findings have implications to the study of automaticity ar.-i chunk- 
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ins. One question to be investigated is if the motor control mechanism is 
central and a modal, as suggested by previous researchers. This can be in- 
vestigated by studying the patterns of interference between modalities. For 
example, an experiment in which the subject is instructed to pronounce one 
character, and write another character at the same time. In addition to pre- 
dicting longer Reaction Time, we can now predict interference between the 
pen stroke patterns and sequences. Another interesting question is how are 
the motoric patterns stored and how are they retrieved when needed. Our 
conjecture, which is consistent with the neural net model, is that the retrieved 
time will be independent of the number of patterns sequences. Some sup- 
port for this conjecture is that it takes the same time to write a character 
in a large character set (kanji) or small character set language (Hiragana, 
English). 

Future research that will combine behavioral analysis with neurobiolog- 
ies! research, might answer many of the questions that we raised in the 
introduction. 
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#include <conio.h> 
#include <stdio.h> 
#include <signal.h> 
#include <math.h> 
#include <stdlib.h> 
#include <graphics.h> 
#include <dos.h> 
#include <string.h> 
#include <values.h> 
#include <iostream.h> 
#include <alloc.h> 
#include <time.h> 
#include <ctype.h> 
#include "qjib.h" 
#include "start.h" 

#define DMAX 400 

#define PERIOD_CONST 10 

#define T 0.01 

#define NUMBER.CHANNEL 5 
#define del 20 

pos.st POS_ST[DMAX]; 

FILE *point,*fp; 

int pen_up=0; 

int init_board (void); 

extern unsigned _stklen=0x8000; 

int read_acc (char *filename); 

void ini_msg(void); 

void next_point(void); 

void read_smooth_point(int number); 

void read_averag_point(float xyz[],long num); 

void get_acc(void); 

#ifdef cplusplus 

typedet void (*fptr)(int); 
#else 

typedef void (*fptr)(); 
#endif 

unsigned int shift_cnt=0; 
int Catcher(int *reglist) 
{ 

printfCCaught it!\n D ); /* make return AX = 3 */ 
return 0; 

} 
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/* First filteration by Butterworth digital filter 4'th order and 
0.1 cutoff freq. 

Input : array of accel. POS_ST and index of point indl. 
Output : smoothed values in array POS_ST */ 
void butter_filt4(pos_st POS_STfl,int indl) 

{ 

static float a[5]={1., -3.1 80639,3.861 194,-2.11 21 55,.438265}; 

static float b[5]={.00041 6599..001 666397..002499595..001 666397..00041 6599}; 

static float y[4][5],x[4][5]; 

int i,j; 

float sum=0.; 
static int cnt=0; 
forG=0;j<4;j++) 

{ 

if ( cnt<5 ) 

y[j][4-cnt] = POS_ST[ind1].AR[j]; 
for(i=4;i>0;i~) 

x[j][i] = xQ][i-1]; 
xD'](0] = POS_ST[ind1].AR[j]; 
if ( cnt >= 5 ) 

{ 

sum = 0; 
for(i=4;i>0;i-) 

y[j][i] = y&'][i-i]; 

for(i=0;i<5;i++) 

sum += b[i]*xfj][i]; 
for(i=1;i<5;i++) 

sum -= a[i]*yfj][i]; 
yDJIO] = sum; 

POS_ST[ind1].AR[j] = sum; 

} 

} 

if ( cnt < 5 ) cnt++; 
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float DIF[2]={0,0},DIFO[2]={0,0}; 

r Second filteration by Butterworth digital filter 4'th order and 
0.02 cutoff freq. 
Input : X and Y accel. 

Output : smoothed values in global variable DIF 7 
void butter_filtO(float dif[2],int fl) 
{ 

static double 
a[5]={1.e+00,-3.83582554064735e+00,5.52081 91 36622236+00,-3.53353521 946302 
e+00,8.48555999266478e-01 }; 

static double 
b[5]={8.984861 46372335e-07,3.59394458504525e-06,5.39091 688001 037e-06,3.59 
394458371 298e-06,8.984861 468164246-07}; 
static float y[2][5],x[2][5]; 
int i,j; 

float sum=0.; 
static int cnt=0; 
if (cnt = 5 && fl) 

cnt=0; 
forO=0;j<2;j++) 
{ 

if ( cnt<5 ) 
{ 

for(i=0;i<5;i++) 
{ 

yQ][i] = difffl; 
xD3[i] = difffl; 

} 

} 

for(i=4;i>0;i--) 

xfflli] = xQ][i-1]; 
x[j][0] = difffl; 
if ( cnt >= 5 ) 

{ 

sum = 0; 
for(i=4;i>0;i~) 

YUM = yO)[i-i]; 

for(i=0;i<5;i++) 

sum += b[i]*x[j][i]; 
for(i=1;i<5;i++) 

sum -= a[i]*yQ][i]; 
yD][0] = sum; 
DIF[j]= sum; 

} 

} 

if ( cnt < 5 ) cnt=5; 

} 
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/* Function for compensation of second filter delay, 
make delay for outputs of first filter. 
Input:d1 and d2 current difference of signals in a pair 
Output: delayed difference signal */ 
void set_delay(float d1 .float d2 t float &res1 .float &res2) 
{static h_cnt=0,t_cnt=0; 
static float diff_acc[2][50]; 
diff_acc[0][Lcnt]=d1; 
diff_acc[1][t_cnt]=d2; 
if ( (h_cnt-t_cnt+del)%del == del-1 ) 
{ 

res1 = diff_acc[0][h_cnt]; 
res2 = diff_acc[1][h_cnt]; 
h_cnt = (h_cnt+1) % del; 

} 

t.cnt = (t_cnt+1) % del; 

} 

struct send_data 
{int period; //T 
float ampl; //A 
float incr_acc; IN 

}; 

send_data params[2][DMAX]; 

/* This procedure gets as inputs two signals (after the filteration) for each 

accelerometer and the pen status. The procedure performs segmentation of 
the signals and calculation of each segment's features V 

void get_pack_param(send_data params[2][DMAX]) 

{static int fLopt=0,cnt_seg[2]={0 I 0}; 

static int cnt_t[2]; 

static float p_dacc[2]={0,0},beg_dacc[2]={0,0}; 
static float max_ampl[2]={0,0} f beg_xyacc[2]={0 f 0}; 
int i; 

float dacc[2]; 

set_delay(POS_ST[1].AR[0]-PEN.R[0]-POS_ST[1].AR[1] + PEN.R[1], 
POS_ST[1].AR[2]-PEN.R[2]-POS_ST[1].AR[3] + PEN.R[3], 
dacc[0] f dacc[1]); 
if (!pen_up) 
{ 

for (i=0;i<2;i++) 
{ 

if ( fLopt ) 
{ 

if { dacc[i]*p_dacc[i] <= 0. ) 
{ 

params[i][cnt_seg[i]].period=cnt_t[i]; 
params[i][cnt_seg[i]].ampl=max_ampl[i]; 
params[i][cnt_seg[i]].incr_acc=DIF[i]-beg_xyacc[i]; 
cnt_seg[i]++; 
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params[i][cnt_seg[i]].period= -1; 

p_dacc[i]=dacc(i]; 

beg_dacc[i]=dacc[i]; 

beg_xyacc[i]=DIF[i]; 

max_ampl(i]=0; 

cnt_t[i]=0; 

} 

else 
{ 

p_dacc[i]=dacc[i]; 

if (fabs(max_ampl[i])<1abs(dacc[i])) 

max_ampl[i] = dacc[i]; 
cnt_t[i]++; 

} 

} 

if (!fl_opt && (fabs(dacc[0])>5. II fabs(dacc[1])>5.)) 
{ 

fl_opt = 1 ; 

cnt_t[0]= cnt_t[1] = 0; 
memcpy(p_dacc,dacc,sizeof(fioat)*2); 
memcpy(beg_dacc,dacc,sizeof(float)*2); 
memcpy(beg_xyacc,DIF,sizeof(float)*2); 

} 

} 

} 

else 
{ 

params[0][++cnt_seg[0]].period= -1 ; 
params[1 ][++cnt_seg[1 ]].period= -1 ; 

} 

} 
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APPENDIX C 



The microcontroller gets as inputs the signals of the accelerometers, performs 
segmentation of it in time (listing of the segmentation program is in appendix B) and 
represent each segment by several parameters which are transmitted to the receiver. 

The segmentation of each signal is performed by the movement of the center 
of oscillations and amplitude and frequency of the oscillations. The procedure of 
segmentation and feature's extraction consists of the next steps: 

Dividing of the original signal into two components: the signal of the movement 
of the center of oscillations and the signal of oscillations. The first signal is 
obtained by filtering the acceleration signal from an accelerometer using 
Butterworth digital filter as described in Digital Filter Design, T.W.Parks and 
C.S. Burrus, John Wiley & Sons, 1987, chapter 7, section 7.3.3, with 4'th order 
and 0.02 cutoff frequency. The second signal is obtained by filtering of the 
differential signal of two accelerometers in a pair, using the Butterworth filter 
v with 4'th order and 0.1 cutoff frequency. 

Segmentation of the signal according to the zeros of the oscillations signal. 
Features extraction - The features that we are using to define each segment 
are: 

Duration between the edges of the segment (T). 
Amplitude of oscillations (A). 

The difference between the values of first signal at the edges of 
segment (V) 



50 



WO 94/18663 PCT/US94/01095 

Fax Pen 

Generator of str^ es for hand imaging 

^include <stdio.h> 
# include <alloc.h> 
# include <math.h> 
# include <string.h> 
#ihclude <graphics.h> 
^include <process.h> 
^include "st.h" 

int far arr xl [15000] ,arr_yl [15000] ; 

void sendjT ( unsigned , char * , unsigned ) ///sending from 
corn-port 

int init com ( int , unsigned ) ; t 

void segment (int num_p, unsigned int far p_x[] , unsigned int far 

]j>_y[] ,char *p) 

extern float xmax,ymax; 

extern shiftx; 

extern char *arg_str[]; 

int v_x[2] [1000] ; 

int v_y[2] [1000] ; 

int extrxy [2] [1000] ; 

int attr=0; 

// structure of binary file of data from graphic tablet 
struct point 

{ 

unsigned x :13; 
unsigned y :13; 
int pen : 6 ; 

// structure of binary file of coefficients of the cubic spline 
struct stroke 
{ 

unsigned x :13, 

unsigned y :13, 

signed alfl:llj 

signed alf2:ll i 
} strk; 
char buff [20] ; 

int j ; 
int newp; 
int num_points; 
int m; 

float lastx,lasty; 
int lvx,lvy; 
FILE *f pla- 
int count , i ; 

unsigned long size_file = 
int fl; 

int lpoint , f point ; 
unsigned base; 
char *pl; 

char *exten={" .etr") ; 
int ascii=0; 
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/* test for extension ".str" .If true - it is the ASCII format 
* for creating a stokes file , i 

strlwr (arg_str [2] ) ; 

if ( (strstr <arg_str [2] , exten) ) ! =NULL) ascii=l ; 
pl=p; 

newp=num_p; 
for(j=0; j<15000;j++) { 
arr_xl [j] =0;arr_yl tj] =0; 

/* V 

* writing binary file with data from graphic tablet * 

* */ 
if (tfpl = fopen (arg_str[l] , "wb" ) ) ! =NULL) 

^ for (count=0;count<num_p ;count++) 
{ 

int p; 
if <P_ X [count] <shif tx) 

pnt.x= (unsigned) p_x [count] ; 

r 1 ' 

else 

pnt.x= (unsigned) (p_x [count] -shiftx) ; 
P=0; 

pnt ,y= (unsigned) p_y[ count] ; 
pnt.pen=p; 

f write (&pnt,sizeof (pnt) , l,fpl) ; 

} 

fseek ( fpl , OL , SEEK_END ) ; 
size_file = ftell ( fpl ) ; 
sprintf (buff , " %lu" , size_f ile ) ; 
setcolor (13) ; 
settextstyle (0,0,1); 
outtextxy (168,40, buf f ) ; 

f close (fpl) ; 

/i — - — */ 

lpoint=fpoint=0; 
fl = l; 

while (nump>lpoint) 

for (i=fpoint ;p_x [i] >=shif tx && i<num__p; i + ; 

attr= (unsigned) (p_x [i-1] -shiftx) ; 

for (j=i;p_x[j]<shiftx j<num_p; j++) 

arr_xl [j -i] «= (f loat)p_x[j] ; 
arr_yl [j-i] = ( float )p_y[j] ; 

num_point s= j - i ; 
newp=num_points ; 
f point=lpoint=j ; 
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/***********************/ 

if (j<num_p) 

{ 



/+ 

* process segmentation * 

* */ 



lastx=arr_xl [num_points-l] ; 
lasty=arr_yl [num_points-l] ; 
extrxy[0] [0] =arr_xl [0] ; 
extrxyll] [0] =arr_yl [0] ; 



j=0; • , • * 

for (i=0;i<=newp-l;i++) 

{ arr_yl t j ] =arr_yl [i] ; 

arr_xl [ j] =arr_xl [i] ; 

if (arr_yl[i] ! =arr_yl [i+1] ) 

arr_xl li] !=arr_xl [i+1] ) 

else 

num_points- - ; 

} 

newp=num_points ; 

/* */ 

lvx=arr_xl [num_points-l] -arr_xl [num_points-2] ; 
lvy=arr_yl [num_points-l] -arr_yl [num_points-2] ; 
v_x[0] [0] » (arr_xl [1] -arr_xl [0] ) ; 
v_y [0] [0] = (arr_yl [1] -arr_yl [0] ) ; 
j=0;m=0; 

/* r */ 

for (i=0 ; i<newp-l ; 1++J 
{ arr_yl t j ] =arr_yl [i] ; 
arr_xl [j] =arr_xl [i] ; 

if (arr_yl [i] !=arr_yl [i+1] 

BcSc arr_xl [i] !=arr_xl [i + 1] ) 

else 

num_points-- ; 

j=0; 

newp=num_points ; 
for (i=0;i<newp-l;i++) 
{arr_yl [j] =arr_yl [i] ; 
arr_xl [j] =arr_xl [i] ; 

if (arr_yl [i] ! =arr_yl [i+1] 

SlSc arr_xl [i] !=arr_xl [i + 1] ) 

else 

num points- - ; 

} 

j=0; 

newp=num_points ; 
f or (i=0 ; i<newp-l ; i++) 
{ arr_yl [j] =arr_yl [i] ; 
arr_xl [j] *=arr_xl [i] ; 

if <arr_yl [i] !=arr_yl [i+1] 
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arr.xl [i) !-arr_xl ti+1) ) 

j++; 
else 

num_points- - ; 



/ 



/****************************/ 

newp=num__point b ; 

j-li 

m=0 ; 

set color (13) ; 
' setfillstyle(l,9) ; 



£il leUipse((£io»t)extrxyC0H0).63 9 /x,nax.47 9 -(floa C , e x C rxvUn 

0] *4 00/ymax-28, 1,1) ; 

for (i=0;i<num_points-2;i++) 

if (arr yl[il<arr_ylti + l] « arr_yl [i + D >arr_yl U + 2] ) 

IxtrxylO] [j]=arr_xlti+l3 ; 
extrxy 111 I j ) =arr_yl [i+l] ; 

fillellipse( (float) extrxytO] [j] ^39/x.ax, 479- (float) extrxy tl] [ 

j] M00/ymax-28 ^ 1 , 1) ; +2] _ arr _ xl ^ , . 

v"y 0 j - (arrlyl [i+2] -arr_yl Ujl ) ; 

r 

if (arr^Uil >arr_yl[i + ll fct arr_yl [i+l] <arr_yl [i + 2] ) 

extrxytO] t j] =arr_xl [i+l] ; 
extrxy [D Ijl =arr_yl [i+l] ; 

fillellipse( (float) extrxy 10] [j] ♦639/xmax, 479- (float) extrxy tl] t 

J-y 0 1 3 j ] = (arrIyl[i + 2] -a^_y L i+l] > ; 
vxtl] tj] = (arr xl U+H -arr_xl lJ , 
vly tl] lj] = (arr:ylli + l)-arr_ylti]) ; 



} 



if (arr x'lUKar^xlti.D « arr.xl [i+l] >arr_xl li+2] ) 

IxtrxylO] [j]=arr_xl[i+l] ; 
extrxy [1] [j] =arr_yl U+H ; 

fillellipse( (float) extrxytO] [j] *639/xmax, 479- (float) extrxy 11] t 
j] •™'y^Jfa l $: lmJKl Ii+2 ] -arr_xl ) ; 
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v y [0] [j] = (arr_yl [i+2] -arr_yl [i+1] ) ; 
v_x[l] [j] = (arr_xl [i+1] -arr_xl [i] ) ; 
v_y [1] [j] = (arr_yl [i+1] -arr_yl [i] ) ; 
)++; 



else 

if (arr xl [i] >arr_xl arr_xl [i+1] <arr_xl U+2] ) 

extrxytO] [ j ] =arr_xl [i+1] ; 
extrxyll] [j] =arr_yl [i+1] ; 

fillellipset (float) extrxy[0] [j] *639/xmax, 479- (float) extrxy [1] [ 
j] *400/ymax-28,l,l) ; 

v_x[0] [j]= (arr_xl [i+2] -arr_xl [i+l] ) ; 

v_y 10] [ j] = (arr_yl [i+2] -arr_yl [i+1] ) ; 

v_x[l] [j] = (arr_xl[i+l]-arr_xl[i] ) ; 

v_y[l] [j] = (arr_yl[i+l]-arr_yl[i] ) ; 

} 



} 



extrxy[0] [j]=lastx; 
extrxyll] [j]=lasty; 



fillellipse ( (float)extrxy [0] [j ] *639/xmax, 479- (float) extrxy [1] [ 
j]M00/ymax-28,l, 1) ; 

v_x[l] [j] =lvx; 

v_y [1] [j]=lvy; 

/ */ 

/* 

* Calculating & writing a stroke file in binary or ASCII format* 

*/ 

{ FILE *ff; 
if (fl) 

ff =ascii ? fopen (arg_str [2] , "w" ) : f open (arg_str [2] . "wb" ) ; 
if (ascii) 

f p r i n t f 

(f f ,"(((((<<(({({■((((<(((((<(({(((( < ( (<<((((<(<<((((<<(<(<<<< ( 
{((((((((((<((({(((((\n M ); 

fl-0; 

} 

else 

ff =ascii ? fopen (arg_str [2] , "a" ) : f open (arg_str [2] , "ab" ) ; 
if (j= = 2 extrxy [0] [ 0 ] = = e x t r xy [ 0 ] [1] && 

extrxyll] [0] ==extrxy [1] [1]) 
{int temp; 

temp=j+ (attr<<9) ; 
memcpy (p , fctemp , 2 ) ; 
p+=2; 

if (ascii) 

f printf (f f , "%6d\n" , temp) ; 
else 

fwrite (fctemp, sizeof (j) , 1, f f ) ; 
strk.x= (unsigned) extrxy [0] [0] ; 
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strk.y= (unsigned) extrxy [1] (0] ; 
strk.alf 1=0; 
strk. alf 2=0; 

memcpy (p, fcstrk, sizeof (strk) ) ; 
p+=sizeof (strk) ; 
if (ascii) , ^ 

fprintf ( f f , M % 4 d % 4 d % 6 
%6d\n", extrxy [0] [0] # extrxy[l] [0] ,0,0) ; 
else 

f write (fcstrk, sizeof (strk) , l,ff ) ; 

} 

else 
{int temp; 
temp=j+ (attr<<9) ; 
memcpy (p, &temp, 2 ) ; 
p+=2; 

if (ascii) 

fprintf (ff , n %6d\n",temp) ; 
else 

f write (fctemp, sizeof (j) ,l,ff) ; 
for (m=0 ;m<j -1 ;m++) 
{ float vx , vy , vxl , vyl ,31,311,312, mods , modv , dlx , dly ; 
dlx=extrxy [0] [m] -extrxy [0] [m+l] ; 
dly=extrxy [1] [m] -extrxy [1] [m+l] ; 
vx=v_x [0] [m] ; 
vy=v_y [0] [m] ; 

mods=sqrt (dlx*dlx+dly*dly) ; 
if (mods==0 . 0) mods=l ; 

modv= (f loat) sqrt ( (vx*vx+vy*vy) ) ; 
if (modv==0 .0) modv=l ; 
all= (dlx*vx+dly*vy) ; 
if(all==0.0) all=0.01; 
al2= (-dly*vx+dlx*vy) ; 

all=al2/all; 
/♦ */ 

vxl=v_X [1] [m+l] ; 
vyl=v_y[l] [m+l] ; 

modv= (float) sqrt (vxl*vxl+vyl*vyl) ; 
if (modv==0 .0) modv=l ; 



al=(dlx*vxl+dly*vyl) ; 
if(al==0.0) al=0.01; 
al2=(-dly*vxl+dlx*vyl) ; 
al«=al2/al; 
al2=al; 

/ 

if (m==j-2) al2=0.0; 
if (m==0) all=0.0; 

if (sign(al2) *al2<0.35 sign (all) *all>l) all/=2.0; 

if (sign(all) *all<0.35 sign (al2) *al2>l) al2/=2.0; 

if (sign(all)*all>5.0) all=sign (all) *1 . 0; 

if (sign(al2)*al2>5.0) al2=sign (al2) *1 . 0; 

strk. x= (unsigned) extrxy [0] [m] ; 

strk. y= (unsigned) extrxy [1] [m] ; 

strk.alfl= (signed) (all*1023. 0/5.0) ; 

strk.alf 2= (signed) (al2*1023 .0/5 . 0) ; 

memcpy (p, &strk, sizeof (strk) ) ; 
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p+=sizeof (strk) ; 

fpAntV^ff , "%4d %4d %6d %6d\n" , extrxy 10] [m] , extrxy ll] [m] , 
strk . alf 1 , strk . alf 2 ) ; 

else 

f write (&strk, sizeof (strk) , l,ff) ; 

^ strk. x= (unsigned) extrxy [0] [m] ; 
strk. y= (unsigned) extrxy [1] [in); 
strk. alf 1=0; 
strk.alf2=0; 

memcpy (p, fcstrk, sizeof (strk) ) ; 
p+=sizeof (strk) ; 

fpAnt^^ff , "%4d %4d %6d %6d\n M , extrxy [0] [m] # extrxy [1] [m] , 
strk. alfl, strk. alf2) ; 

else 

fwrite (fcstrk, sizeof (strk) , l,ff ) ; 
f close <ff).; 

^ if (! (base=init_com(2,9600) ) ) exit(-l); 

send_p< base , pi , (unsigned) (p-pl) ); 
if (ascii) 
{int leng; 
char ch[20]={«") ; 
fpl = fopen (arg_str [2] , n a" ) ; 
fseek ( fpl , 0L , SEEK_END ) ; 

( f P f i , -)))))> n ))))))) n )))))))))))))))))))))))))))))))))))))) ) 

>>>))>>>>>>>>)>>\n">; 
'fclose (fpl); 
strcpy (ch, arg_str [2] ) ; 
leng=strlen (ch) ; 
strcpy (&chtleng- 3] , "bst") ; 
fpl = fopen (ch, "wb") ; 
fwrite (pi, (unsigned) (p-pl) ,l # fpl) ; 
fclose (fpl) ; 

fpl = fopen (arg_str [2] , "r") ; 

fseek ( fpl , 0L , SEEK_END ) ; 
size_file = ftell ( fpl ) ; 
sprintf (buf f , B %lu" , size_file ) ; 
setcolor (13) ; 
if (ascii) 

outtextxy (580, 40, buff ) ; 
sprintf (buff , »%lu" ,p-pl ) ; 
outtextxy (368, 40, buf f ) ; 

} 

else 

outtextxy (368, 40, buff ) ; 
fclose (fpl) ; 

free (p) ; 
free (pi); 
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3.1. Definition of coordinate systems : 



PCT/US94/01095 




oyx* Intrinsic (Local) coordinate system describing the pen 
stroke. 

OYX-Extrinsic (Global) coordinate system describing the world 
(e.g computer screen) 
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RErONSTRT irTTON GOAL : 

generation of a sequence of third-order (cubic) splines in local 
coordinate system and transition from the local oyx syster. to the 
global OYX system. 
The directions of the OYX vectors 

that define the segmentation of a symbol are chosen in 
accordance with the OX axes direction (local system) . 




in accordance with fig 2, the information needed for 
reconstruction of a symbol consist of : 



the 





...*.] 






V V . 





where 
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are the angles between the correspc-ding vectors 

LI L- 

and the vector n. 
The vectors 

VI 5& 

of the speed in the verric of the ■ skeleton n of the symbol (in 
the example in fig 2, it is the .--veer "a"). 
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Fax Pen 

pprftnRt n igt ^ on of h and imaging . 

^include <stdio.h> 

# include <conio.h> 

^include <math.h> 

tfinclude <string.h> 

^include <process.h> 

# include <graphics.h> 

ft include "st.h" 

float xmax=913:*5; 

float ymax=594 . *5; 

int per=50; 

float v_x[2] [1000] ; 

float v_y [2] [1000] ; 

int dotted=0; 

unsigned char attr=0; 

int cols[5]={8,9,10,12,14); 

int extrxy[2] [1000]; 

//calculation of splinr function with 2 derivatives and 2 points 
float spl (float , float .float .float float , float , float ); 
void main (int num_arg, char *arg_st[J) 

// structure of binary file of cubic spline coefficients 
struct stroke 
{ 

unsigned x :13; 
unsigned y :13 
signed alf 1 : 11 
signed alf 2 :11 

•in^gdriver = DETECT, gmode, errorcode; 
int j=0; 
int m; 
int i; 
int x; 
FILE *ff; 

float seg,lseg; 

char *exten={" .str" }; 

int ascii=0; 

/* test for file extension ".str" .If true - it is the ASCII 

format * - f 

* for creating stokes rile / 
strlwr (arg st [1] ) ; 

if ( (strstr{arg_st [1] ,exten)) !=NULL) ascn=l; 

if (num arg<=l) {printf ( "Requires 1 parameter"); exit(l);} 

if((ff =ascii ? 

fopen(arg_st [1] , «r») : 1 open (arg_st [1] , "rb» ) ) ==NULL) 
{printf ("File not found"); exit(l);} 

registerf arbgidrivrr (EGAVGA_driver_f ar ) ; 
registerf arbgifont {sansserif_f ont_f ar) ; 
initgraph ( fcgdriver ( fcgmode , " 11 ) ; 
errorcode = graphresult ( ) ; 
if (errorcode != grOk) 
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^ print f ("GraphicB error: %s\n", grapherrormsg (errorcode) ) ; 
print f ( "Press any key to halt:"); 
getch ( ) ; 
exit (1) ; 

setbkcolor (15) ; 
setcolor (1) ; 
rectangle (0,0, 639, 479) ; 
line(0,35,639,35) ; 
setfillstyle(l,8) ; 
bar(l, 1,638,34); 
settext style (3,0,4); 
setcolor (11) ; 

outtextxy(142, -5, "The Stroke Interpreter"); 



if (ascii) {char 
*strl«" (<{(<<<((((<<(<(<((((<<<(<<<<( U ( ( ((((((((((((((((MM ( 

((U((((((((((<(((((((( M ; 

int res=l; 

while (res!=0) 
{ char *str2; 
f scanf (f f , "%78s\n M ,str2) ; 
res=s trcmp ( str 1 , str2 ) ; 

while (1) 
{int temp; 
if (ascii) 

i=f scanf (ff , "%d\n",&temp) ; 
else 

i=f read (&temp, sizeof (j) , 1, f f ) ; 

j^temp Sc OxlFF; 

attr=temp>>9; 
if (i<=0) 
{ 

f close (ff ) ; 

if ( ! getch () ) getch {) ; 
closegraph ( ) ; 
exit (1) ; 
} 

i=0? 

{int vx,vy; 
while { i<j ) 
{ 

if (ascii) 

^ fscanf(ff>"%d %d %d 
%d\n" , fcextrxy [0] [i] , fcextrxy [1] [i] , &vx, &vy) ; 
v_x[0) [i] = ( (float)vx*5. 0/1023.0) ; 
v x[l] [i] = ( (float)vy*5. 0/1023.0) ; 

}" 

else 

(f read (fcstrk, sizeof (strk) , l,ff ) ) ; 
extrxytO] [i] = (signed) strk. x; 
extrxy[l] [i] = (signed) strk.y; 
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/* 

* preparation from integer to float * 

v x[0) [i)«=(etrk.alf 1*5. 0/1023.0) ; 
v~x[l) [i)=(strk.alf2*5. 0/1023.0) ; 

}" 
i++; 

j=i; */ 
/* */ 

setcolor (cols [attr&0x07] ) ; 

setlinestyle (0,0,2* ( (attr&OxOB) »3 ) +1) ; 

dotted= (attr&OxlO) >>4; 

lf( putpixel (extrxy [0] [0] , extrxy [1] [0] , cols [attr&7] ) •; 
else 

{ 

lseg=seg=0; 

//drawing splines 

for (m=0;tn<j-l ;m++) 

{ float y , xl , yl , all , al2 , bete , bets , mods , dlx , dly ; 

dlx=extrxy [0] [m+l] -extrxy [0] [m] ; 

dly^extrxy [1] [m+l] -extrxy {1] [m] ; 

all=v_x[0] [m] ; 

al2=v_x[l] (m] ; 

mods=sqrt <dlx*dlx+dly*dly) ; 

xl^extrxy [0] [m] ; 
yl=extrxy [1] [tn] ; 

• /* -*/ 

/★★★★A*************************** 

* 

* cos and sin of angle between local and global system 
* 

*****************++*++**+*******/ 

betc=dlx/mods ; 
bets=-dly/mods ; 
/ 

for (i=0;i< ( (int ) mods+1) ;i+=2) 
{float x0,y0; 
x= (float) i; 

* calculation of local spline function y=f (x) with 
parameters : 

* (start x & y, finish x & y, angels in start & finish, 

argument x. 
* 

********************************/ 

y=spl (0.0,0.0, mods , 0 . 0 , al2 , all , mods -x) ; 

* 

* rotation of local system to global system 
* 

★*******************************/ 
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xO=betc*x-bets*y+ (float) extrxy[0] [m] ; 
y 0= (-bets*x-betc*y+ (float) extrxy [1] [m] ) ; 
if (dotted) {float dx,dy; 

dx= (float) (xO-xl) ; 

dy= (float) (yO-yl) ; 

dx*=dx; 

dy*=dy; 

seg=sqrt (dx+dy) ; 
lseg+=seg; 

if (Idotted II (((int)lseg)%(2*per))<per) 

line( (float)x0*639/xmax / 479-(float)y0*443/ymax / 
(float) xl*639/xmax, 479- (float) yl*443/ymax) ; 
xl=xO;yl=yO; 

^ if (dotted) "{float dx,dy; 

dx= (float) (extrxy [0] [m+l]-xl) ; 
dy= (float) (extrxytl] (m+l]-yl) ; 
dx*=dx; 
dy*=dy; 

seg=sgrt (dx+dy) ; 
lseg+=seg; 

if (idotted II (<<int)lseg)%(2*per))<per) 

line ( (float)xl*639/xmax,479- (float ) yl*443/ymax, 

(float) extrxy [0] [m+1] *639/xmax, 479- (f loat ) extrxy [1] [m+1] *443/y 
max) ; 

} 

} 

i=i; 

A- - */ 



} 
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CPT.TNE RBrnwfiTRUCTIOK : 

Threshold conditions : (margin conditions) : 

The spline derivatives on the interval [0.11] edges, where 11 
module of the vector LI (see fig 1.) 

y(0) =tco 2 
cccj, co-Oj_are calculated as; 

V.L. Iiv£ll 
cosa. = _ - - ; sine,*-- - ,,- 

to-a 1 =sir.a 1 /coso 1 ; c^Oj=sina./cosa 2 
where [.] vector product defined zs .: 

J-x L y L :t 

whezeV xt V y .V. V L x .L y .L : 
projections of the vectors 

V u 1 

on axes Ox, Oy, Oz according to the significance of the spline 



on the interval (01,11} are 
y(0) = y(H) = 0 
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A spline is defined ir. the form of : 

y{x) *a 1 x 3 *a 2 x 2 ^a 3 x+a 4 

where 

are coefficients that are defined by margin conditions 
The equations system relatively unknown values 

a.,, 1 = 1+4 . 



PCT/US94/01095 



or 



y(0) =a 4 =0 

*(i J )»3a 1 i 1 *+2a 2 Va : -t:pa l 
y(0)=a^tgra 2 





M 















for 

a 4 =0 , a 3 = tga 2 

The transformation to a global systeir. is being done in accordance 



with the expressions: 



(X) { cosPi sinp.V x \ 
W^-sin^ cosplj\y,x)J 



The examples of the different letters reconstruction are given 
on fig 3. 
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#include <conio.h> 
#include.<stdio.h> 
#include <math.h> 
#include <dos.h> 
#include <stdlib.h> 
#include <graphics.h> 
#include "qjib.h" 
#define DMAX 400 

#defme T 0.01 

struct send_data 
{int period; 
float ampl; 
float incr_acc; 

}; 

int reconstruct(send_data par[2],int t_count[2),float prev_dif[2]) 
{ 

static float px=-10000,py=-10000; 
float D!F[2]; 
int i; 

float acc[4]; 
float u1; 
float u2; 
float shift[2]; 

const float a1 1=1 .071 524; 
const float a1 2=0. 11965; 
const float a21 =0.075; 
const float a22=-0.22502; 
const float b1 1=1 3.333333; 
const float b1 2=1 3.333333; 
const float b21 =6.666667; 
const float b22=6.666667; 

{ 

float dx; 
float dy; 
float x,y; 
setcolor(15); 

/* Restoration of oscillation signal */ 
u1=par[0].amprsin(t_count[0]*M_PI/par[0].period); 
u2=par[1 j.ampr sin(t_count[1 ]*M_PI/par[1 j.period); 

/* Restoration of the movement of the center of oscillations */ 
DIF[0l=t_count[0]/par[0].period+prev_difI0]; 
DIF[1]=t_count[1]/par[1].period+prev„dif[1]; 
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/* reconstruction of the position of the pen's tip */ 
dx=(airu1+a12*u2); 
dy=(a21*u1+a22*u2); 
shift[0] = (birDIF[0]+b12*DIF[1]); 
shift[1] = (b2rDIF[0]+b22*DIF|1]); 
if (px> -10000 && py> -10000) 
{ 

// Addition of the movement of the center of oscillations to obtain XY coordinates. 
x=100+(dx+shift[0])*2; 
y=200+(dy+shift[1])M; 
'f(i) 
{ 

line(px,py,x,y); 

} 

// Saving coordinates of previouse point 
px=x; 

py=y; 

} 

else 
{ 

px=100+(dx+shift[0])*2; 
py=200+(dy+shift[1])*4; 

} 

} 

return 0; 
} 
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APPENDIX I 

The reconstruction procedure is performed in two stages: 

- Restoring the signals. 

- Reconstruction of the position of the pen's tip. 

The restoration of the acceleration's signals from the data that was transmited 
by the pen (T,A,V for every segment) is done according the next formula: 
u x (U) = u x0 +u x1 ; 
u x0 =u x (i-1J )d . l ) + {V x /T xi )M; 
u x1 = A xi * SIN ((Pl/T^) * t) ; 

u y (U) = u^ +u y1 ; 
Uyo^UyO-U^ + ^/T^M; 
u y1 = V S,N «PI/T„)M); 

u,(tf) = +u 2l ; 

u z0 = u z (MJ zM ) + <V 2 /T ri ) M ; 

u„ = A* * SIN ((PlfTJ * t) ; 

u x ,u y ,u z are the restored signals of accelerometers. 

Uj^Uy^u^ are the restored movement of the center of oscilations. 

u x1 ,u yt ,u z1 are the restored signal of oscilations. 

The reconstruction of the position of the pen's tip in XY plane is done by 
decomposition of two movements: the movement of the center of oscilations in XY 
plane (x 0 ,y 0 ) and the oscilation movement (contour of written symbol) (x^y,). The 
calculation of these values is according the next formulas: 

X 0 = a i1* U xO + a i2 U y0 
Yo = a 21 *U x0 + a 22*"y0 

x t = b 1t *u x1 + b 12 *u y1 
Yi = b 21 *u x1 + b 22 *u y1 

The parameters , b^ vary from individual to individual and are received as the 
personal hand imaging characteristics of the writer at the beginning of a session. 
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C LA IMS 

1; Communication apparatus for band imaging comprising: ' 

apparatus for sensing features of hand imaging of an 
individual which features are highly characteristic of the indi- 
vidual but which also contain information relating to images 
represented thereby; and 

apparatus for providing a non-individual dependent 
output indicating the images in response to the sensed features. 



2. Communication apparatus according to claim 1 and where- 
in said apparatus for sensing features is contained in a hand- 
held housing. 

3. Communication apparatus according to claim 1 and where- 
in said apparatus for sensing features is contained in a tablet 
assembly. 



4. Communication apparatus according to claim 1 and also 
comprising apparatus for communication of the non-individual 
dependent output. 

5. Communication apparatus according to claim 2 and also 
comprising apparatus for communication of the non-individual 
dependent output. 

6. Communication apparatus according to claim 3 and also 
comprising apparatus for communication of the non-individual 
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dependent output. 

7. Apparatus according to claim 4 and wherein said appara- 
tus for communication is operative to communicate information 
which can be used to reconstruct an individual's hand imaging 
style. 

8. Apparatus according to claim 2 and wherein said sensing 
apparatus does not require a tablet. 

9. Apparatus according to claim 4 and wherein said appara- 
tus for communication comprises a modem. 

10. Apparatus according to claim 4 and wherein said appara- 
tus for communication is operative to communicate in a fax for- 
mat. 

11. Apparatus according to claim 4 and wherein said appara- 
tus for communication is operative to communicate in a compressed 
non-raster format. 

12. Apparatus according to claim 4 and wherein said appara- 
tus for communication is operative for wire communication. 

13. Apparatus according to claim 4 and wherein said appara- 
tus for communication is operative for wireless communication. 

14. Apparatus according to any of the preceding claims and 
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wherein said apparatus for sensing features includes apparatus 
for sensing the instantaneous angle of motion during hand imag- 
ing. 

15. Apparatus according to claim 14 and wherein said 
apparatus for providing a non-individual dependent output is 
operative for providing an output indication of strokes generated 
during hand imaging. 

16. Communication apparatus for hand imaging comprising: 
apparatus for sensing features of hand imaging of an 

individual which features are highly characteristic of the indi- 
vidual but which also contain information relating to images 
represented thereby; and 

apparatus for providing an output indicating the images 
in response to the sensed features, and wherein 

said apparatus for sensing features includes apparatus 
for sensing the instantaneous angle of motion during hand imaging 
and providing an output indication of strokes generated thereby. 

17. Apparatus for communicating hand imaging comprising 
hand-held apparatus for sensing motion and providing an output in 
a compressed form which can be transmitted by a conventional 
modem, LAN or other communications medium. 

18. Apparatus according to any of claims 1-13 and 16-17 and 
also comprising apparatus for receiving communicated stroke 
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content information and being operative for reconstructing there- 
from hand-imaging information. 

19. Apparatus according to claim 14 and also comprising 
apparatus for receiving communicated stroke content information 
and being operative for reconstructing therefrom hand-imaging 
information. 

20. Apparatus according to claim 15 and also comprising 
apparatus for receiving communicated stroke content information 
and being operative for reconstructing therefrom hand-imaging 
information. 

21. Apparatus according to claim 18 and wherein said appa- 
ratus for receiving is operative to reconstruct hand-imaging 
information in three dimensions. 

22. Apparatus according to claim 19 and wherein said appa- 
ratus for receiving is operative to reconstruct hand-imaging 
information in three dimensions. 

23. Apparatus according to claim 20 and wherein said appa- 
ratus for receiving is operative to reconstruct hand-imaging 
information in three dimensions. 

24. Communication apparatus for hand imaging including: 
apparatus for sensing motion during hand imaging and 

providing an output indication of stroke content in a compressed 
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format; and 

apparatus for receiving communicated stroke content 
information and being operative to reconstruct therefrom hand- 
imaging information. 

25. Apparatus according to claim 24 and wherein said appa- 
ratus for receiving is operative to reconstruct hand-imaging 
information in three dimensions. 

26. A communication method for hand imaging comprising: 
sensing features of hand imaging of an individual which 

features are highly characteristic of the individual but which 
also contain information relating to images represented thereby; 
and 

providing a non-individual dependent output indicating 
the images in response to the sensed features. 

27. A communication method for hand imaging comprising: 
sensing features of hand imaging of an individual which 

features are highly characteristic of the individual but which 
also contain information relating to images represented thereby; 
and 

providing an output indicating the images in response 
to the sensed features, and wherein 

said sensing features includes sensing the instantane- 
ous angle of motion during hand imaging and providing an output 
indication of strokes generated thereby. 
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28. A method for communicating hand imaging comprising 
hand-held apparatus for sensing motion and providing an output in 
a compressed form which can be transmitted by a conventional 
modem, LAN or other communications medium. 

29. A communication method for hand imaging including: 
sensing motion during hand imaging and providing an 

output indication of stroke content in a compressed format; and 

receiving communicated stroke content information and 
reconstructing therefrom hand-imaging information. 
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